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A Preliminary Examination of Baltimore Ingenuity Student Outcomes: 

Classes of 2008 and 2013 

Executive Summary 

The Ingenuity program was designed to “provide Baltimore’s brightest middle school students 
with a free, highly accelerated, and challenging mathematics and science curriculum” (Ingenuity 
Project, 2014). It started in 1993 at two middle schools, one on the east side of Baltimore and the 
second on the west, but as of SY 2014-15 the program is in place in several middle schools and 
one high school. Students must participate in a competitive application process, and those 
selected represent some of Baltimore’s brightest and most motivated students, a group who some 
worry is too often ignored in urban school districts. Participants benefit from an extraordinary 
opportunity for enriched, accelerated math and science course taking and mentorship. 

Students served by Ingenuity are less likely to be African American or receive free/reduced price 
meals than the district in general. The program serves approximately 270 middle school students 
(approximately 90 students in each grade) and 120 in high school (around 30 in each grade per 
year). Ingenuity Project is offered at Baltimore Polytechnic Institute (Poly) and in the middle 
grades at Hamilton, Mt. Royal, and Roland Park K-8 schools. 

Research questions for this analysis were developed using a participatory model that included 
staff from the Ingenuity Project, Baltimore City Schools, the Abell Foundation, and university 
researchers of BERC. Questions are primarily focused on whether the program has an impact on 
high school and postsecondary outcomes, as well as student self-efficacy. The analysis features a 
comparison with a set of similar students who never participated in Ingenuity. It also includes 
current and fonner participants’ responses to an online survey about their career plans and 
progress. 

The findings show that a cohort of students who participated in the Ingenuity program during 
middle school only out-performed comparable peers in terms of high school academic behaviors 
(e.g., advanced course and AP exam-taking), and outcomes (e.g., course grades or SAT scores). 
Students who participated in the high school component also showed significantly higher 
outcomes relative to similar peers who did not participate, and were also more likely than 
comparable students to have completed a four-year college degree after four years. 

According to current Ingenuity students’ survey responses, over 95% intend to pursue a four- 
year degree, and approximately three-fourths are interested a STEM field of study. Likewise, 
about 95% of former Ingenuity students reported they were enrolled in a four- year degree- 
granting college, with two-thirds studying for a STEM career. Respondents from both groups 
most commonly were interested in, or were pursuing studies in biology and/or medicine. Among 
alumni participating in the survey who are now enrolled in college, virtually all are making 
expected (or accelerated) progress towards a degree. 

There are several cautions that must be made with respect to the study’s generalizability. We 
could not control for unobserved characteristics among the Ingenuity students and the 
comparison groups that represent important differences that may have affected outcomes, 
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especially parental support for education, socioeconomic background and the intrinsic qualities 
of the students themselves, such as enthusiasm for mathematics and science. 

Most importantly, however, the non-ingenuity students may not have had the same course-taking 
opportunities as Ingenuity participants. Students at some middle and high schools had access to 
more rigorous course options, especially in math and science, which made true comparisons 
impossible. 

As a result, we avoid making any causal statements about the effects of Ingenuity, as students 
were not randomly assigned to the program but were targeted for participation because of 
exceptional qualifications. 
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A Preliminary Examination of Baltimore Ingenuity Student Outcomes: 

Classes of 2008 and 2013 

Rachel E. Durham 


Background 

The Ingenuity program is intended to “provide Baltimore’s brightest middle school students with 
a free, highly accelerated, and challenging mathematics and science curriculum” (Ingenuity 
Project, 2014). Started in 1993, two participating middle schools were strategically selected to 
host Ingenuity based on their location, with one each in east and west Baltimore to ensure equal 
geographic access. Since that time, the program has expanded to one high school, Baltimore 
Polytechnic Institute, and is now offered at several middle schools different from the inaugural 
schools. 

Students must first complete a challenging application process to participate in Ingenuity. 
Selection is based on a review of students’ past grades, state assessment scores, daily attendance, 
and teacher recommendation letters, as well as additional standardized testing. Students’ 
application portfolios are scored, and those with scores in the top percentile ranking among all 
applicants are invited to participate (percentile cut-points depend upon both demand and available 
seats for the upcoming school year). Students chosen for this program benefit from an 
extraordinary opportunity for enriched, accelerated math and science course taking and 
mentorship. They represent some of Baltimore’s brightest and most highly motivated students, a 
group who some worry is too often ignored in urban school districts. 

Ingenuity serves a population of students that is different from the district overall. This is true 
even within Baltimore Polytechnic Institute (Poly). During SY 2012-13, 58% of students 
attending Poly received FARM services and 76% were African-American and/or Hispanic, 1 
whereas among Ingenuity graduates of 2013, 41% received FARM and 50% were ethnic 
minority. This difference suggests that the opportunity to benefit from Ingenuity is concentrated 
among students who are, on average, more affluent and less ethnically diverse than their school 
peers. 

Moreover, Ingenuity serves a relatively small share of Baltimore’s students. Around 90 students 
in each of sixth, seventh, and eighth grade participate in the Ingenuity middle grades program 
(i.e., approximately 270 per school year, or slightly less than 2% of all City Schools 6 th -8 th 
graders). Currently three schools serving grades 6 through 8 offer Ingenuity - Hamilton 
Elementary/Middle, Roland Park Elementary/Middle, and Mt. Royal Elementary/Middle School. 
Typically, only about one-third of these are accepted to continue into the high school Ingenuity 
program at Poly. As a result, each new cohort of ninth grade Ingenuity participants includes 
around 30 students. Appendix A provides a timeline showing the opening and closing dates of 
each school’s Ingenuity program. 


1 Figures from Poly for 2012-13 were obtained from mdrcportcard.org | retrieved May 25, 20 1 4 1 . 
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Methodology 

This section will review the research questions, and describe the data and procedures used to 
respond to each question. 

Research Questions 

This report will respond to the following research questions, using de-identified administrative 
data records provided by the Office of Achievement and Accountability at City Schools: 

1) Relative to a comparable set of peers, did middle grades-only participants: 

• Earn more advanced math and science course credits? 

• Achieve higher grades in math and science courses, as well as overall? 

• Take more Advanced Placement (AP) courses, a greater number of AP exams, 
and did they receive higher scores on the exams? 

• Achieve higher SAT and PSAT scores? 

• Graduate on time at a higher rate? 


2) Relative to a comparable set of peers, did high school graduates of Ingenuity: 

• Earn more advanced math and science course credits? 

• Achieve higher grades in math and science courses, as well as overall? 

• Take more Advanced Placement (AP) courses, take a greater number of AP 
exams, and did they receive higher scores on the exams? 

• Achieve higher SAT and PSAT scores? 

• Enroll more often at postsecondary institutions, and at more selective 
institutions (per Barron’s Profiles of American Colleges, 2009)? 

• Complete more four- year degrees? 

3) Do current and former Ingenuity students report strong efficacy in STEM, and 
attachment to STEM-related career goals? 


Cohort Analysis 


Three cohorts of students were included in the quantitative analysis of high school outcomes: 

• Those who participated in Ingenuity only during grades 6-8 from 2006-07 through 2008- 
09 and did not continue into the high school Ingenuity program, but whose on-time 
graduation year was 2013, 

• Those who participated in Ingenuity at Baltimore Polytechnic Institute in 9 th - 12 th grades 
beginning in 2004-05 and graduating in 2007-08, 

• Those who participated in Ingenuity in Baltimore Polytechnic Institute in 9 th - 12 th grades 
beginning in 2009-10 and graduating in 2012-13. 
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Middle Grades-Only Participants. The focal group for this portion of the analysis included 49 
students who participated in Ingenuity during grades 6 - 8 in 2006-07 through 2008-09, 
respectively. Comparison students: a) were in fifth grade in 2005-06, b) never participated in 
Ingenuity during grades 6 through 12, and c) attended at least one year of high school in 
Baltimore City so an outcome status could be determined. Both comparison and Ingenuity 
middle grades students would have been on-time graduates in 2012-13. 

High School Participants . In 2012-13 there were 28 Ingenuity graduates, and in 2007-08 there 
were 24. For each Ingenuity cohort, a set of 17 comparison students were identified. The 
comparison students: a) started ninth grade at Poly in the same year as the Ingenuity cohort, b) 
had never participated in Ingenuity themselves, and c) remained in a City Schools high school 
through graduation. Virtually all Ingenuity and comparison graduates attended and graduated 
from Poly. 

Identical matching procedures were used to identify comparison students for both the middle and 
high school outcome analyses. We matched students according to gender, ethnicity, FARM 
status, LEP status, special education service receipt, prior attendance rate, and middle grades 
state assessment scores in math and reading. (More details about Ingenuity students’ and the 
comparison groups’ characteristics, as well as the matching procedure can be found in 
Appendices B and C, respectively.) 

Surveys 


To answer the last research question concerning Ingenuity participants’ self-efficacy and 
attachment to science/technology/engineering/math (STEM) fields of study, we solicited 
responses to two online surveys: one targeted to current high school Ingenuity participants who 
were in 10 th , 11 th , or 12 th grade during SY 2013-14, and a second for former Ingenuity students 
who, as of spring 2014, were one, two, or three years past high school graduation. Participants 
for the current student survey data collection effort were recruited by Ingenuity staff working in 
the high school with current program students, and all Ingenuity students in grades 10 through 12 
were given the option to participate. For the alumni survey data collection effort, participants 
were recruited through email inquiries from Ingenuity staff, who have attempted to maintain 
contact information for their graduates. However, given that former students who continue to be 
in contact with Ingenuity staff may have a more favorable perception of the program, the 
respondents who participated in the survey (as well as those for whom email addresses were 
even valid and available) may represent a select population of Ingenuity alumni. 

The majority of the questions used in the survey were taken from the Pre-College and Current 
Study versions of the Assessing Women and Men in Engineering Annual Self-Efficacy Surveys 
(2006), which were developed under a National Science Foundation grant to examine students’ 
trajectories through STEM courses of study. A second source for the survey questions pertaining 
to perceptions of self-efficacy was the Cooperative Institutional Research Program’s (CIRP) 
Freshman Survey (2014) (used with permission). Using previously-developed survey items is 
advantageous, since their relationship to underlying constructs of self-efficacy and attachment 
has already been determined to be valid and reliable. 
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The survey response rate was approximately 70% for current Ingenuity students and 56% for 
Ingenuity alumni. 2 Questions asked about basic demographics, their plans for STEM-related 
careers, and for ratings of agreement with items gauging self-confidence, self-efficacy, and their 
expectations around a career in STEM. 


2 No non-ingenuity students were surveyed for comparison purposes, due to the infeasibility of identifying 
appropriate controls ex ante of knowing the characteristics of survey respondents. Further, all survey participation 
was anonymous and no personally identifying information was collected. 


Ingenuity Program 


4 


Baltimore Education Research Consortium 


Findings 

The next sections will respond to each research question beginning with the analysis of outcomes 
for the middle school-only participants, followed by an outcome analysis for the high school 
Ingenuity program participants, and finally we present the results of the surveys administered to 
current Ingenuity participants and recent alumni. 


Middle Grades-Only Participants’ Outcomes 

Earning Credits in Advanced Math and Science . To measure differences for middle grades-only 
participants in advanced course taking, we first identified City Schools courses that are 
considered to be “advanced” with guidance from the Office of Gifted and Advanced Learning. 
For math, such courses offer credits beyond those required for Algebra I/Data Analysis, 
Geometry and Algebra II, and include AP Calculus AB/BC, AP Statistics, and Honors 
Trigonometry, among others. For science, advanced courses include those beyond Biology I or 
Environmental Science I, Chemistry I, and Physics I. Examples of these courses included 
Microbiology I, AP Biology, AP Chemistry, AP Physics, Thermodynamics, Honors Research 
Practicum, etc. 

Concerning the number of “advanced” credits subsequently earned in high school math and 
science, we found that students who had been in Ingenuity only in grades 6 through 8 earned, on 
average, 3.5 credits in math compared to 2.5 among comparison students (see Table 1). For 
advanced science credits, Ingenuity students earned 3.9, as compared to 2.7 among their peers. 
Both differences were statistically significant. 

Table 1 

Mean Credits Earned in Advanced Mathematics and Science during High 
School, for Ingenuity Middle Grades Cohort and Comparison 

Ingenuity - Middle Comparison - No 
Grades Only Ingenuity 

3.5** T5 

3.9* 2/7 

**p<.01 *p<.05 


Adv. Math 
Adv. Science 


Grades in Math and Science . When we compared performance in advanced as well as regular 
math and science high school courses, students who had been Ingenuity participants earned 
higher grades, and all differences were statistically significant except for advanced math course 
performance. Ingenuity students also had significantly higher weighted GPAs at the end of high 
school than comparable students, with an average 3.0 compared to 2.5 (see Table 2). It should 
be noted that weighted GPA reflects a weighting factor of 1.2 for any AP course and a factor of 
1.1 for an Honors course; thus if the middle school Ingenuity participants had more opportunities 
to take AP or honors courses in high school than students in the comparison group, then average 
GPA would yield a greater difference in performance than numerical course grades. 
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Table 2 


High School Course Performance, for Ingenuity Middle 

Grades Cohort and Comparison 

Ingenuity - 

Middle Grades Comparison - 

Only No Ingenuity 


Advanced Math 80.3 76.6 

Advanced Science 81.6** 75.1 

All Math 81.3* 77.1 

All Science 81.9** 76.5 

Weighted Final GPA 3.0** 2.5 

**p<.01 *p<.05 


AP Courses, Exams and Scores . As seen in Table 3, Ingenuity students earned significantly 
more Advanced Placement credits in high school than comparison students (9.3 versus 5.8), and 
they took more AP exams and received higher AP scores, although neither difference was 
significant. 


Table 3 


Advanced Placement Credits, Examinations and Average Scores, for 
Ingenuity Middle Grades Cohort and Comparison 



Ingenuity - 
Middle Grades 

Comparison - No 


Only 

Ingenuity 

Credits Earned (N) 

9.3 f 

5.8 

Exams Taken (N) 

2.0 

1.4 

Average AP Exam Score 

2.7 

2.2 


**p<.01 *p<.05 J p<.10 


SAT and Preliminary SAT (PS AT) Table 4 contains average PSATs and SATs that represent a 
student’s highest score reported for a subject, from any testing administration. This distinction is 
important for students who took assessments two or more times, as their highest math score 
might have been achieved during a testing session that did not produce their highest verbal or 
writing score, and further, their highest score may not be their most recent. Then, the highest 
combined score (sum of verbal, math and writing sub-scores) is the highest combined score they 
achieved during any single testing session. 

With respect to the average highest PSAT and SAT scores they achieved, we found that middle 
grades-only Ingenuity students later achieved significantly higher scores than comparison 
students, across all subject areas and for both exams. 
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Table 4 

Mean SAT and PSAT performance, for Ingenuity Middle Grades 
Cohort and Comparison 
Ingenuity - 

Middle Grades Comparison - No 
Only Ingenuity 

PSAT Performance 


Verbal 

56** 

48 

Mathematics 

55** 

48 

Writing 

54** 

47 

Percent who took PSAT 

98% 

96% 

SAT Performance 

Verbal 

574** 

498 

Mathematics 

564** 

487 

Writing 

559** 

484 

Combined 

1679** 

1451 

Percent who took SAT 

98% 

90% 


**p<-01 *p<.05 


On-Time Graduation. Finally, nearly all middle grades-only Ingenuity students graduated on 
time in the spring of 2013, though one was in 12 th grade but did not graduate that year. Among 
the comparison group, 90% graduated on time, a handful of students were in 12 th grade but did 
not graduate that year, one student had fallen behind in grade, and one student had dropped out 
of school (see Table 5). 


Table 5 


High School Completion Outcomes, for Ingenuity Middle Grades Cohort 

and Comparison 


2012-13 Graduation Status 

Ingenuity - 
Middle Grades 
Only 

Comparison - 
No Ingenuity 

% Graduated 

98.0 f 

89.8 

% Not Graduate, but in 12 th grade 

2.0 

6.1 

% 11 th grade 

0.0 

2.0 

% Dropped Out 

0.0 

2.0 


**p<.01 *p<.05 f p<.10 


Ingenuity Program 


7 


Baltimore Education Research Consortium 


High School Participants’ Outcomes 

These analyses include students who participated only during high school or were in the program 
during both middle school and high school. Too few students participated in either the middle 
and high school or high school-only program for two separate analyses, so both groups were 
considered together against similar comparison peers. 

Taking Advanced Courses in Science and Math. Table 6 presents the mean number of credits 
earned by high school Ingenuity participants and their comparison peers in advanced math and 
science courses. In all instances, the mean difference was statistically significant with Ingenuity 
students earning more advanced credits than comparable peers. An interesting finding is that 
relative to 2007-08 Ingenuity graduates, the more recent 2012-13 graduates had earned more 
credits in both math and science. 


Table 6 

Mean Credits Earned in Advanced Mathematics and Science, for 
Ingenuity High School Cohorts and Comparison 



2007-08 

2012-13 


Ingenuity 

Comparison 

Ingenuity 

Comparison 

Adv. Math 

2 9 ** 

2.2 

4.1** 

2.6 

Adv. Science 

4.7* 

3.5 

7 2 ** 

2.5 


**p<.01 *p<.05 


Grades in Math and Science . Final weighted GPA comparisons were possible for 2013 
graduates, but were not available for the 2008 graduates due to recent changes in district grading 
policy. Conversations with district research personnel suggested that replicating the current 
procedure for the older cohort would be difficult if not impossible, since course offerings, 
descriptions, and weighting policies have changed since 2008. Thus, for the 2008 graduates, we 
only compare final numerical grades. This limitation also suggests that comparisons of the 
courses of study between Ingenuity graduates in 2008 and 2013 should be made with caution. 

Table 7 


High School Course Performance, for Ingenuity High School Cohorts and 
Comparisons 


Courses 

2007-08 

2012-13 

Ingenuity 

Comparison 

Ingenuity 

Comparison 

Advanced Math 

87.3** 

78.1 

85.2* 

79.1 

Advanced Science 

90.2** 

83.5 

86.3** 

76.1 

All Math 

88 . 1 ** 

81.0 

86.2 

81.2 

All Science 

89.0** 

83.3 

85.8** 

77.2 

wGPA, all Courses 

- 

- 

3.5** 

2.7 


**p<.01 *p<.05 


Table 7 shows that for both comparisons of advanced math and science courses, high school 
Ingenuity students earned significantly higher marks than comparable students. Then, concerning 
all math and science course taking (which includes advanced as well as required courses) we 


Ingenuity Program 


8 


Baltimore Education Research Consortium 


found that among the 2008 graduates, Ingenuity students earned significantly higher grades. For 
the 2013 graduates, Ingenuity students received significantly higher marks in science, but not 
math, as compared to similar peers at Poly. However, when weighted GPA for all courses taken 
in high school was considered, the average GPA of Ingenuity participants was significantly 
higher at 3.5, compared to 2.7 for their peers. Though as seen in Table 6 , the 2013 Ingenuity 
graduates had earned more advanced credits, and many advanced courses have AP and honors 
designations, which are weighted more heavily in GPA calculations. 

AP Course Taking in Science and Math . Ingenuity graduates in both cohorts earned more 
Advanced Placement credits and took more than twice as many AP exams as comparable 
students. As seen in Table 8 , performance on AP exams varied by cohort. 

Table 8 

Advanced Placement Credits, Examinations and Average Scores, for Ingenuity 


High School Cohorts and Comparison 



2007-08 

2012-13 


Ingenuity 

Comparison 

Ingenuity 

Comparison 

N, AP Credits Earned 

5.4** 

2.1 

7.5** 

2.0 

N, AP Exams Taken 

6.4** 

2.3 

7 3 ** 

2.0 

Average AP Exam Score 

3.3* 

2.3 

3.0 

3.1 


**p<.01 *p<.05 


These differences between both cohorts and their comparison peers with respect to the number of 
AP exams taken may also indicate disparities in course-taking opportunities for Ingenuity 
students at Poly, relative to non-ingenuity students. 

Ingenuity graduates from the Class of 2008 scored significantly higher on AP exams than their 
comparison peers, while those from the later cohort, on average, earned scores that were 
statistically similar to those of their peers, though Ingenuity students took over twice as many 
exams. 


SAT and Preliminary SAT (PSAT) . Table 9 contains average PSATs and SATs representing a 
student’s highest score reported for a subject, from any testing administration. (As before, their 
highest score may not be their most recent.) Unfortunately, PSAT scores for the years that the 
2008 graduates would have been tested (i.e., fall 2006 and 2007) were not available. 

In each instance considered, Ingenuity students achieved significantly higher PSAT and SAT 
scores than comparable students. Further, Ingenuity students who graduated in 2013 were more 
likely to have taken the SAT than comparison students. Among the 2013 graduates, for whom 
PSAT comparisons can be generated, it appears that Ingenuity students showed more 
improvement than their peers between taking the PSAT and sitting for the SAT. Calculating the 
predicted SAT score from a PSAT score is achieved by multiplying the PSAT score by 10, so for 
instance, among the comparison graduates of 2013, the average PSAT verbal score, 52, was 
close to the SAT verbal scores they eventually earned, 533. But among Ingenuity participants, 
their average SAT verbal score of 619 was much higher than the predicted 580 from their 
average PSAT verbal score of 58. 
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Table 9 


Mean SAT and PSAT performance, for Ingenuity High School Cohorts and Comparison 


2007-08 

2012-13 


Ingenuity 

Comparison 

Ingenuity 

Comparison 

SAT 





Verbal 

647** 

581 

619** 

533 

Mathematics 

693* 

636 

684** 

583 

Writing 

650* 

592 

602** 

523 

Combined 

1971** 

1791 

1887** 

1611 

Percent who took SAT 

100% 

92% 

100% f 

89% 

PSAT 





Verbal 

- 

- 

58* 

52 

Mathematics 

- 

- 

65** 

55 

Writing 

- 

- 

56* 

49 

Combined 

- 

- 

179** 

155 

Percent who took PSAT 

- 

- 

100% 

100% 


**p<.01 *p<.05 f p<.10 


College Enrollment and Institutional Selectivity (per Barron’s Profiles of American Colleges, 
2013) . At this time, postsecondary enrollment data are available only for the 2008 graduates. 
National Student Clearinghouse (NSC) data report enrollment dates and the institutions where 
students enrolled, and the most recent City Schools’ NSC data are capable of providing this 
information for the graduating classes of 2005 through 2012. While we believe any 
postsecondary experience is noteworthy, for the current research question we elected to consider 
enrollments that occurred during the fall immediately following high school graduation. This 
approach allows us to consider student differences demarcated at a single point in time, and 
furthermore, enrollments following on the heels of graduation signal a commitment to the 
postsecondary study path whereas delayed enrollment may suggest that students are considering 
career options that do not require a college degree. 

Table 10 


Fall Enrollments at Postsecondary Institutions among the Class of 2008, 
Ingenuity High School Cohort and Comparison 



Ingenuity 

Comparison 

Fall Enrolled 

83.3% 

91.7% 

Two-Year College 

0.0% 

6.3% 

Four- Year College 

100.0% 

93.8% 

Selectivity: 

Very Selective 

55.0% 

29.4% 

Selective 

30.0% 

23.5% 

Somewhat Selective 

15.0% 

23.5% 

Not Selective 

0.0% 

23.5% 


Note. No significant differences on any measure. Source: NSC. 
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Table 10 presents a comparison of the percentage of Ingenuity graduates and their peers who 
enrolled at a college or university in the fall after graduation. Although a higher share of 
comparison than Ingenuity students enrolled the following fall, 3 this difference was not 
statistically significant. Further, all of the fall enrollments by Ingenuity participants were in 
four- year institutions, and the schools they attended had higher selectivity ratings, per Barron’s 
profiles (Barrons, 2013). But again, none of these differences were significant. 

Earning College Degrees . Data from NSC also provides information on degree completion. The 
most current NSC data represented a time window that would have allowed the 2008 graduates 
four years plus one fall semester to complete a bachelor’s degree. As shown in Table 1 1, a 
higher share of Ingenuity participants had completed a degree during this time frame. This is 
true for those who enrolled in the fall following high school, as well as when considering 
enrollments at any time after high school. 

While comparisons of degree completion were only marginally statistically significant, the 
differences are certainly substantively significant given degree completions were on the order of 
one and a half times greater among Ingenuity graduates. 

Table 11 

Postsecondary Degree Completion among the Class of 2008, 

Ingenuity High School Cohort and Comparison 

Ingenuity Comparison 

(N=24) (N=17) 

62.5% 41.7% 

75.0% 45.5% 


All Students: 

Among only Fall-Enrolled: 


Both differences were marginally significant (p<.08). Source: NSC. 


Efficacy in STEM and Attachment to STEM-related Career Goals 

To gauge current and former Ingenuity students’ self-perceptions around STEM and their career 
plans and progress, we solicited responses to two online surveys: one targeted to high school 
Ingenuity participants who were in 10 th , 11 th , or 12 th grade during SY 2013-14, and a second for 
Ingenuity alumni who, as of spring 2014, were one, two, or three years past high school 
graduation. As noted previously, survey response rates were around 70% for current Ingenuity 
students and 56% for Ingenuity alumni. 

As shown in Table 12, the high school survey respondents were 51% male, half were 10th 
graders, one-third were in 12th grade, while around 41% were white and 30% African-American. 
A little more than half reported that at least one of their parents had an advanced/graduate 
college degree. 


3 In analyses not shown, it was found that if college enrollments at any time were considered, Ingenuity graduates 
enrolled at a higher rate than their comparison peers, though this difference was not statistically significant. 
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Table 12 

Description of Current Ingenuity Students 
Completing the Survey 

Grade Levels: 

Percentage 

10 th 

49.4 

11 th 

18.8 

12 th 

31.8 

Male 

51.0 

Female 

49.0 

White 

40.7 

African-American 

30.2 

Hispanic 

3.5 

Asian 

5.8 

More than one 

11.6 

No Answer 

8.1 

Parents’ Education Level: 

High School or less 

10.5 

Some college 

11.6 

College Graduate 

19.8 

Advanced Degree 

51.2 

No Answer 

7.0 

Number of Respondents 

Who Finished Survey 

83 


When asked about post-high school plans, about three-fourths of current high school Ingenuity 
students reported that they intend to go to college to prepare for a career in STEM. Most 
respondents were “fairly sure” or “very sure” about their career intentions, and nearly all 
reported a specific field of study in which they were most interested. 


Table 13 

Career Plans of Current High School Ingenuity Students 

Planning for a STEM Career 

Percentage 

77.6 

No plans for a STEM Career 

22.4 

How confident are you about your plans? 

Not very confident; it’s likely to change 

4.7 

50% chance I’ll change 

9.4 

Fairly confident 

52.9 

Very confident 

32.9 
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Table 13 

Career Plans of Current High School Ingenuity Students 

Percentage 

What Field(s) of study are you interested in pursuing? 

Medicine/Biology 

32.6 

Physical/Mech/Elec Engineering/Physics 

25.0 

Social Science/Statistics 

12.0 

Chemistry/Chemical Engineering 

9.8 

Computer Science/Software Design 

6.5 

Environmental Science/Geology 

4.3 

Astronomy/Astrophysics 

4.3 

Forensic Science 

3.3 

Humanities/Fine Arts 

2.2 


What sources of information have you used to make your 
plan regarding career and college? 


Parent(s) 

77.9 

Ingenuity peers 

54.7 

Campus visits 

53.5 

Ingenuity teachers 

48.8 

High school activities 

48.8 

Counselors 

48.8 

Other college activity(s) 

45.3 

Other family members 

34.9 

Other high school teachers 

34.9 

Employer(s) 

8.1 


The range of fields in which students reported wanting to pursue a degree are presented in Table 
13. Students most frequently reported interest in medicine or another field within biology, 
followed by physics and engineering. Specifically, students gave responses such as, 
biotechnology, biomedical engineering, biochemistry, neurology, zoology, neonatology, 
molecular biology, genetics, or simply “medicine” or “biology.” After biological fields, there 
was common interest in electrical engineering, mechanical engineering, civil engineering, or just 
“physics.” The next most popular career interests were in the social sciences (sociology, 
economics, psychology, public health), followed by chemistry and chemical engineering. 

Many students listed more than one interest, though most students seem to have narrowed their 
interest to a single field of science but were still considering more than one particular focus 
within that field. 

Most students reported more than one source of information about going to college, with about 
three-fourths reporting their parents, and one-third other family members. More than half 
reported that they got information from college visits, their peers in the Ingenuity program, and 
nearly half from Ingenuity teachers, college activities, high school activities, or counselors (see 
Table 14). 
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When asked how hard they believed they work to get good grades in their high school classes, 
and how hard they expected they will need to work in college, 43% reported that they “work 
hard” in their high school courses while about one-third said they just “work some, but not too 
hard.” A handful of students said it was “easy”, or “easy with a few exceptions.” However, 63% 
reported they expect that they will need to work harder when they are in college. Further, most 
expect that they will adjust well to college life, and over 80% were confident in their ability to 
complete a college degree (see Table 14). 


Table 14 

Current Ingenuity High School Students’ Expectations about College 

Percentage 

How easy is it to get the grades you want in high school classes? 


It’s easy 3.5 

With a few exceptions it’s easy 14.0 

I work some, but not too hard 36.0 

I have to work hard 43.0 

I don’t work hard; don’t care about grades 3.5 

How hard do you think you’ll have to work in 
college? 

I’ll have to work less 8.1 

I’ll have to work about the same 29. 1 

I’ll have to work harder 62.8 

To what extent do you agree that you’ll adjust well to college? 
Strongly Disagree 0.0 

Disagree 1.2 

Slightly Disagree 0.0 

Neither agree nor disagree 4.7 

Slightly Agree 7.1 

Agree 41.2 

Strongly Agree 45 . 9 

How sure are you that you’ll complete a college degree? 

Not sure, highly likely I won’t finish 1 .2% 

About 50% sure 2.3% 

Fairly confident 16.3% 

Very confident 80.2% 


Next, students were asked about their feelings of belonging within the fields of STEM, their 
expectations for success in a STEM-related career, their expectations concerning personal 
satisfaction if they pursue a career in STEM, self-efficacy with respect to STEM, as well as for 
self-assessments of their knowledge, creative thinking skills, and soft skills (see Appendix C for 
specific questions and constructs). 

As seen in Table 15, most current high school Ingenuity participants reported agreement (“agree” 
or “strongly agree”) with statements expressing an expectation for success and personal 
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satisfaction in a STEM career. They also expressed a high degree of self-efficacy around STEM 
course performance and their ability to achieve balance between professional and personal lives. 

On average self-assessments of knowledge and creative thinking were high, but interestingly, 
girls rated themselves significantly lower than boys on confidence in their ability to think 
creatively. We also noted that those who declared an intention to pursue a career in STEM were 
significantly more likely to agree with statements relating to feelings of belongingness within 
STEM, expectations for satisfaction and success with a STEM career, and STEM-related self- 
efficacy. 


Table 15 


Current Ingenuity High School Students’ Mean Self-Ratings on Efficacy, Confidence, 
Expectations and Belongingness around STEM 



Feeling of Expectations 
Belonging for Personal 
within Satisfaction 
STEM with a Career 
Field in STEM 

Expecta- 
tions for 
Successful 
Career in 
STEM 

STEM 

Self- 

Efficacy 

Self-Rating 

of 

Knowledge 
Generally 
and in your 
Field 

Self- 

Rating of 
Ability to 
think 

Creatively 

Self- 
Rating 
of Soft 
Skills 

Grade 








10 th 

5.3 

5.4 

6.0 

6.1 

4.2 

4.3 

3.0 

H th 

5.2 

5.4 

6.4 

6.3 

3.9 

4.3 

2.8 

12 th 

5.5 

5.4 

6.0 

5.9 

4.1 

4.3 

2.8 

Gender 








Male 

5.3 

5.5 

6.1 

6.2 

4.2 

4.5 

2.9 

Female 

5.3 

5.3 

6.0 

6.0 

4.1 

4.1** 

2.9 

STEM 

Career 








Have Plan 

5.6 

5.8 

6.3 

6.3 

4.1 

4.4 

2.9 

No Plan 

4 4** 

3 9 ** 

5.2** 

5.2** 

4.1 

4.2 

2.9 

All 

5.3 

5.4 

6.1 

6.1 

4.1 

4.3 

3.0 


**p<.01 

Note . For the first 4 constructs, answers could range from l=Strongly Disagree to 7=Strongly Agree. 


For last 3 constructs, answers could range from l=Major Weakness to 5=Major Strength. 


Ingenuity Alumni. About one-third of the Ingenuity alumni who completed the survey graduated 
City Schools in 2011, 20% graduated in 2012, and 45% in 2013. The distribution of alumni by 
their parents’ highest educational level and race/ethnicity was similar to that for current 
Ingenuity students, with around 40% white and 36% African American and around 48% having a 
parent with an advanced degree. Slightly more male than female alumni participated in the 
survey. 
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Table 16 

Description of Ingenuity Alumni Completing 

the Survey 

Percentage 


Graduation year: 


2011 

34.1 

2012 

20.5 

2013 

45.5 

Male 

59.1 

Female 

40.9 

White 

40.9 

African-American 

36.4 

Hispanic 

2.3 

Asian 

11.4 

More than one 

6.8 

No Answer 

2.3 

Parents’ Education Level: 

High School or less 

18.2 

Some college 

15.9 

College Graduate 

13.6 

Advanced Degree 

47.7 

No Answer 

4.5 

Number of Respondents 

Who Finished Survey 

44 


When we asked about their current status and career plans, 42 of the 44 Ingenuity alumni 
reported that they were enrolled in a four- year college, one reported being enrolled in a two-year 
college, and one reported that s/he was neither in college nor working. About two-thirds 
reported that they were working towards a four- year degree within a STEM field, while one-third 
were studying non-STEM subjects. In either case, more than 90% of respondents were fairly or 
very confident about their plans. 

Alumni reported their progress toward a degree, and we compared this to their year of high 
school graduation (see Table 17). All of the alumni enrolled in college were in a stage of their 
studies that corresponds with expected progress or were even further along in credit accrual. 

Table 17 


Spring 2014 Matriculation Status (%) of Ingenuity Alumni, by Year 
of High School Graduation 


Year 

Graduated (N): 

College 

Freshman 

College 

Sophomore 

College 

Junior 

College 

Senior 

2011 (14) 

0 

0 

60.0 

40.0 

2012 (9) 

0 

77.8 

11.1 

11.1 

2013 (20) 

55.0 

45.0 

0 

0 
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Alumni reported a more narrow range of fields of study than those reported by current high 
school Ingenuity students (which would be expected for those already accruing credits in a 
particular field). But as with current Ingenuity students, most reported medicine or another area 
of biology. The second most frequently reported field of study was physics or engineering, 
followed by computer science or software development (see Table 18). 

Most alumni reported that they felt they had had enough information about career options during 
their high school years to make an informed decision about a college major (not presented in 
tables), with 68% agreeing or strongly agreeing; however, about one-fourth of alumni disagreed 
that they’d had enough information about career choices when they made a decision concerning 
where to attend college and what their major would be. 

When asked to imagine themselves in 20 years and the sort of organization in which they would 
want to be working, about one-fifth of alumni reported working in a non-academic research 
organization, nearly one-fifth in an academic environment, and almost one-sixth pictured 
themselves starting their own enterprise. About one-third reported other and among these, one 
reported ‘financial analyst,’ one stated ‘working in a church,’ another said ‘US Navy,’ and one 
pictured himself working for the U.S. government, while a handful reported that they imagined 
themselves as some type of clinical medical provider. Interestingly, none of the former 
Ingenuity students said they expected to be a school teacher, and a handful reported that they 
didn’t yet know in what arena they would be working. 


Table 18 

Percent of Ingenuity Alumni by Reported Career Plans 

Percentage 


Preparing for a Career in STEM 65.9 

No plans for Career in STEM 34. 1 

How Confident are you about your plans? 

Not very confident; it’s likely to change 4.5 

50% chance I’ll change 4.5 

Fairly confident 59.1 

V ery confident 31.8 

What Field(s) of Study are you pursuing? 

Medicine/Biology 45.5 

Phy sical/Mech/Elec Engineering 21.2 

Computer Science/Software Design 15.2 

Social Science/Statistics 9.1 

Environmental Science/Geology 6.1 

Astronomy/Astrophysics 3.0 

Forensic Science 
Chemistry/Chemical Engineering 
Humanities/Fine Arts 
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Table 18 

Percent of Ingenuity Alumni by Reported Career Plans 

Percentage 

In 20 years, in what arena do you expect you’ll be working? 


An academic environment, like a professor 6.8 

An academic environment, mostly applied research 1 1.4 
Private enterprise that I build myself or with a team 15.9 
A non-academic organization where I am paid to do 
research or development 20.5 

Teaching in a public or private school 0.0 

I don’t know yet 15.9 

None of the above; something different 29.5 


Measures of efficacy, attachment to STEM and other competencies, showed no significant 
differences between male and female respondents, or according to the stage of their college 
studies (results not shown). Not surprisingly, alumni who reported they were preparing for a 
career in STEM had significantly higher scores than those who reported they were not pursuing 
STEM, on both their expectations for a personally rewarding career in STEM, as well as STEM- 
related career success. This comparison along with average ratings across all respondents are 
reported in Table 19 below. 


Table 19 


Mean Self-Ratings on Efficacy, Confidence, Expectations and 
Belongingness around STEM for Ingenuity Alumni 


Plans 

Feelings of 
Belonging 
within 
STEM 
Field 

Expectations 
for Personal 
Satisfaction 
with a 
Career in 
STEM 

Expectations 

for 

Successful 
Career in 
STEM 

STEM 

Self- 

Efficacy 

Self-Rating 

of 

Knowledge 
Generally 
and in your 
Field 

Self-Rating 
of Ability 
to think 
Creatively 

Self- 

Rating of 
Soft Skills 

STEM Career 

5.6 

5.9 

6.3 

6.3 

4.3 

3.3 

4.0 

No plans 

5.0 

49 ** 

5.6* 

5.8 

4.1 

3.5 

3.8 

All 

5.4 

5.6 

6.1 

6.2 

4.2 

3.3 

3.9 


**p<-01 *p<.05 

Note . For the first 4 constructs, answers could range from l=Strongly Disagree to 7=Strongly Agree. For 
last 3 constructs, answers could range from l=Major Weakness to 5=Major Strength. 
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Discussion and Conclusions 

This report examined whether Ingenuity students outperform similar peers with respect to high 
school and postsecondary outcomes. Analyses focused on two groups: students who participated 
in only the middle school program and those who participated in either the high school or the 
middle and high school components of Ingenuity. Outcomes included rigor and performance in 
high school, college preparation, enrollment in college, and degree completion. In addition, we 
surveyed current and former Ingenuity participants about their attachment to STEM fields of 
study and work. 

The Ingenuity program serves a small number of students annually, about 90 students each in 6 th , 
7 th and 8 th grade, representing less than 2% of students in those grades across the district. In high 
school, about 30 students are served in each grade, representing less than 0.5% of high school 
students in the district. 

The analysis of Ingenuity student outcomes revealed that middle school-only participants earned 
more advanced math and science credits, more AP credits, achieved higher GPAs, higher PSAT 
and SAT scores, and were more likely to graduate on time from high school, as compared to 
students who were never in Ingenuity. Notably, 94% of those who had participated in Ingenuity 
during the middle grades were enrolled in the entrance criteria schools of Poly, Dunbar, City 
College, Baltimore School for the Arts, or Western High School, whereas only 71% of the 
comparison group attended these 5 schools. This difference suggests the potential for inequities 
in the opportunities students had for taking advanced math and science courses, AP courses, and 
other extracurricular activities that may have increased their engagement with school and driven 
their high school outcomes, as program and course offerings across high schools in Baltimore 
vary tremendously. 

Another related caveat is that while some of the students captured in the comparison group for 
the middle grades-only portion of the analysis were attending the same middle schools as the 
Ingenuity participants, 65% attended different middle schools, and thus may have experienced 
highly /^comparable levels of academic rigor to prepare them for high school. 4 

High school Ingenuity participants from both the Class of 2008 and 2013 graduating cohorts 
outperformed comparable students in terms of the number of challenging courses for which they 
earned credit, the number of AP exams they took, their scores on AP tests and SAT tests, as well 
as the grades earned in their high school courses. While Ingenuity graduates of 2008 enrolled 
the following fall in postsecondary institutions at around the same rate as comparable peers, they 
had finished four- year degrees at nearly twice the rate during the four and a half years since 
graduation. However, these results should be interpreted with caution, since the Ingenuity 
students likely had different middle school experiences than those in the comparison groups, 
which may have strongly affected their later outcomes. In fact, four- fifths of the 2012-13 
Ingenuity graduates at Poly had attended either Roland Park or Mt. Royal in middle school 
(among those enrolled in City Schools in 8 th grade), whereas their comparable peers were 
distributed across 10 other different middle schools. 


4 We explored limiting the potential comparison group to students enrolled at the two participating Ingenuity middle 
schools only, but given the scope of the program at these schools, too few non-ingenuity students remained for 
this approach to he feasible. 
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Another limitation to these analyses is that a comparison group identified according to several 
demographic characteristics and performance on a single state assessment does not necessarily 
constitute a comparable group with respect to other unobserved characteristics, especially 
intrinsic academic motivation, interest in math and science, and socioeconomic advantage. 

These unobserved qualities also affect performance and may account for differences in outcomes 
observed between Ingenuity and comparison students. 

Identifying appropriate comparison students within Baltimore, especially for rising 9 th graders 
chosen to participate in Ingenuity as high school students, was extremely challenging. This is 
evidenced by the fact that only 17 comparable students could be identified for either graduating 
cohort. This indicates that Ingenuity’s selection process successfully achieves its goal of 
enlisting the most academically equipped students into its program (at least in terms of 
standardized testing skills, which is a large part of the selection process). Very often, Ingenuity 
students’ average MSA or Terra Nova scale scores were higher than the individual scores of 
many “advanced” students who were not Ingenuity participants. 

In the survey component, the majority of current Ingenuity students and recent alumni reported 
they have plans for (or are studying towards) a career within a STEM field, most often medicine 
or engineering. Around 95% of the alumni reported they were enrolled in a four-year degree- 
granting college or university, and over 95% of current Ingenuity students reported intentions to 
pursue a four- year degree. The vast majority of both groups reported a high degree of self- 
efficacy and feelings of belonging within STEM, and strong expectations for successful and 
personally rewarding STEM careers, although female current participants rated themselves 
somewhat lower than males on their ability to think creatively. 

While the response rate for Ingenuity alumni was lower than desired for making firm statements 
about this group, two-thirds of those who completed the survey remain strongly attached to goals 
involving math and science. Moreover, most of the alumni respondents have made notable 
progress toward these goals through their postsecondary studies. 

At the time of this writing, City Schools is developing a district-wide portfolio of gifted and 
advanced learning programs, within which the Ingenuity Project is envisioned as an important 
but small part of a new, more comprehensive set of offerings. The findings above point to several 
related implications: 

• Systemic, district-wide opportunities for advanced learning among elementary- aged 
students are needed to nurture the gifts and talents of the youngest served by City 
Schools. A program to identify students with special interests and abilities early on can 
increase student and family engagement with schooling for years to come. 

• While it is notable that the Ingenuity middle-grades program is offered at three schools in 
geographically and demographically diverse sections of Baltimore, greater equity in 
access to advanced learning options for the middle grades is needed. In particular, 

Algebra I is a gateway course to many advanced math course taking opportunities in high 
school, and accessing these is key to students’ subsequent postsecondary options. City 
Schools should consider the feasibility of offering Algebra I at all schools serving 8 th 
graders. 
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• Students at all of City Schools’ high schools should have opportunities to earn credits in 
AP and honors courses. Expanding such options across the district would better enable 
its students to submit competitive applications to colleges. College applicants are 
adjudicated according to their exposure to rigorous courses, as well as GPA, which can 
be strongly impacted by performance in advanced coursework. 
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Appendix A 

Timeline of Schools’ Participation in the Ingenuity Program 


Garrison Middle 
(2003 - 2005) 


Hamilton E/M (2008- ) 


Mt. Royal E/M (2005 - ) 


Roland Park E/M (2003 - ) 


Baltimore Polytechnic Institute (1997 - ) 


7 




Robert Poole Middle (1994 - 2004) 


Southeast Middle (1994-2005) 


7 


> 


West Baltimore 
"Middle (1994-1996) 


1993 


1998 


2003 


2008 


2013 
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Appendix B 

Matching Results for Ingenuity and Comparison students 

Middle Grades-Only Participants. The focal group for this portion of the analysis included 49 
students who participated in Ingenuity during grades 6 - 8 in 2006-07 through 2008-09, 
respectively. 5 Identifying a suitable set of students to serve as a comparison group was an initial 
step. To do so, we first selected into the potential comparison pool students who: a) were in fifth 
grade in 2005-06, b) never participated in Ingenuity during grades 6 through 12, and c) attended 
high school in Baltimore City so that an outcome status could be determined. Both comparison 
and Ingenuity middle grades students would have been on-time graduates in 20 12- 13. 6 

Once potential matches were defined, a logit matching procedure 7 was estimated. Essentially, 
the probability of being an Ingenuity participant was regressed against the student variables: 
ethnicity, gender, 5 th grade FARM services, special education services, LEP status, 5 th grade 
attendance rate, and 5 th grade MSA scores in reading and math. From this estimation, we were 
able to identify 49 comparison students who were determined to be, on average, comparable to 
the 49 Ingenuity participants across all matching variables. 

Table B.l presents means for each matching characteristic by group, and the data show that they 
were adequately balanced with no significant differences on any measure. It is notable, however, 
that administrative records show that 13 of these comparison students who were not participating 
in Ingenuity were enrolled in Roland Park Elementary/Middle School, and 3 were attending Mt. 
Royal Elementary/Middle in 2008-09 (their 8 th grade year). The remaining 33 comparison 
students’ middle school enrollments were distributed across the district in 20 different schools. 

Table B.l 


Means for Characteristics Used in Matching Procedure for Ingenuity 
Middle Grades and Final Comparison Students 



Ingenuity - 
Middle 

Comparison - 


Grades Only 

No Ingenuity 

% Male 

28.6 

28.6 

% Free/Reduced-price lunch 

41.0 

51.0 

eligible 
% White 

16.3 

12.2 

% African-American 

77.6 

85.7 

% Asian 

4.1 

2.0 

% Hispanic 

2.0 

0.0 


5 There were originally 90 students identified as Ingenuity middle grades participants, but 29 continued into the high 
school Ingenuity program and were not appropriate to include in this part of the analysis. Further, an additional 12 
students did not attend a City School high school between 2009-10 and 2012-13, and thus data that would allow us 
to measure their high school performance were unavailable. 

6 Participant rosters submitted to City Schools by Ingenuity did not contain information for the graduating class of 
2007-08 that would allow the identification of Ingenuity middle grades -only participants during their 6 ,h -8 ,h grade 
years, i.e., SYs 2001-02 through 2003-04. Rosters included only information pertaining to 2008 Ingenuity graduates 
who followed the program into high school. The resulting pool of Ingenuity middle grades participants included 49 
students, among whom 30 attended Roland Park Elementary/Middle School while 19 attended Mt. Royal 
Elementary/Middle School. 

7 Further details about the propensity score matching procedure can he found in Appendix C. 
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Table B.l 

Means for Characteristics Used in Matching Procedure for Ingenuity 
Middle Grades and Final Comparison Students 

Ingenuity - 

Middle Comparison - 

Grades Only No Ingenuity 


5 th Grade Attendance Rate 

97.9 

98.1 

5 th Grade MSA Math Scale Scores 

Composite 

460.6 

460.3 

Algebra patterns/functions 

510.8 

507.2 

Geometry measures 

452.1 

451.1 

Statistics and Probability 

495.1 

505.6 

Number 

512.1 

521.0 

Relations/Computations 

Math Processes 

460.0 

457.6 

5 th Grade MSA Reading Scale 
Scores 

Composite 

451.9 

448.5 

General reading 

450.2 

444.0 

Information reading 

453.6 

452.3 

Literary reading 

455.3 

452.6 

N 

49 

49 


Note . No significant differences for any measure. For characteristics where the 
means were different, the variance within either student group was greater than the 
mean difference between the groups. 


High School Participants. Just as for middle grades-only participants, an initial step was to 
identify appropriate comparison students for the high school Ingenuity participants. This was 
done separately for each of the 2008 and 2013 Ingenuity graduating classes. We defined a pool 
of potential matches for each group by selecting students who: a) started ninth grade at Poly in 
the same year as the Ingenuity cohort, b) had never participated in Ingenuity themselves, and c) 
remained in a City Schools high school until they withdrew or graduated so that outcomes could 
be determined. 

Using an identical matching procedure as for identifying suitable comparison students for the 
middle school component, we matched students according to gender, ethnicity, FARM status, 
LEP status, special education service receipt, 8 th grade daily attendance rate, and middle grades 
state assessment scores in math and reading. From this estimation, a set of comparison students 
was identified for each cohort of high school Ingenuity participants. 

As shown in Table 2, the matching technique produced sufficiently balanced groups of 
comparison students for further analysis. For the 2012-13 graduating cohort comprising 28 
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Ingenuity graduates, 8 17 comparison students were identified. For the 2007-08 cohort, which 
included 24 Ingenuity graduates, a different set of 17 comparison students was identified. 
Virtually all graduates of Ingenuity and their corresponding comparison students attended and 
graduated from Poly. 9 

Across the variables used to define similarity, few differences between groups remained and 
none were significant. As compared to all other prospective peers entering Poly in 9 th grade 
(results not presented), the students included as comparison cases for both cohorts were far less 
likely to qualify for FARMS, had very good attendance in 8 th grade (missing fewer than 4 days 
on average), and achieved scores on their 7 th grade Terra Nova and 8 th grade MSA assessments 
that placed virtually all of them above the 90 th national percentile, and in the advanced category, 
respectively. 


Table B. 2 


Means for Characteristics Used in Matching Procedure for High School Ingenuity 

and Final Comparison Students 



2007-08 

2012-13 


Ingenuity 

Comparison 

Ingenuity 

Comparison 

% Male 

62.5 

70.8 

75.0 

75.0 

% Free/Reduced-price lunch eligible 

16.7 

16.7 

50.0 

46.0 

% White 

33.3 

50.0 

29.0 

21.0 

% African-American 

50.0 

37.5 

46.0 

61.0 

% Asian 

12.5 

0.0 

11.0 

7.0 

% Hispanic 

4.2 

12.5 

4.0 

11.0 

% American Indian 

0.0 

0.0 

11.0 

0.0 

8 th Grade Attendance Rate 

98.2 

98.1 

98.2 

98.4 

7 th Grade Terra Nova Math Scale 
Scores 





Composite 

739.4 

737.1 



Math Computation 

742.7 

743.8 



Math Concepts & Applications 
7 th Grade Terra Nova Reading Scale 

735.9 

730.3 



Scores 





Composite 

718.1 

717.7 



Reading Comprehension 

706.9 

713.7 



Reading Vocabulary 

728.9 

721.2 



8 th Grade MSA Math Scale Scores 





Composite 



481.3 

484.0 

Algebra patterns/functions 



500.0 

504.8 


8 One Ingenuity student from the 2012-13 cohort was excluded from all analyses due to a near-zero probability of 
identifying an appropriate control or set of control students. 

9 One comparison student for the 2007-08 cohort transferred out of Poly before the end of his/her junior year in high 
school but graduated from City Schools in the expected amount of time. The remaining comparison students for 
both graduating Ingenuity cohorts were enrolled at Poly for the entirety of their high school years. 
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Table B. 2 


Means for Characteristics Used in Matching Procedure for High School Ingenuity 

and Final Comparison Students 



2007-08 

2012-13 


Ingenuity Comparison 

Ingenuity 

Comparison 

Geometry measures 


523.6 

527.6 

Statistics and Probability 


503.6 

493.7 

Number Relations/Computations 


512.6 

519.2 

Math Processes 


474.4 

473.8 

8 th Grade MSA Reading Scale 




Scores 




Composite 


441.3 

445.4 

General reading 


439.4 

437.8 

Information reading 


453.8 

461.8 

Literary reading 


440.8 

451.6 

N 

24 17 

28 

17 


Note . No significant differences for any measure. For characteristics where the means were different, 
the variance within either student group was greater than the mean difference between the groups. 
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Appendix C 

Methodological Details for Propensity Score Matching Procedure 

The propensity score method used for the current study was performed to identify comparison 
(i.e., “control”) subjects who, on average, were as similar as possible to subjects who received 
the “treatment,” in this case - Ingenuity participation. Ideally, control subjects must not at any 
point in time have received the treatment themselves, and any characteristics used to “match” 
subjects should be measured prior to the start of the treatment. 

To account for the first condition, any student identified as an Ingenuity participant according to 
roster data provided by City Schools was excluded from the pool of potential control subjects. 

For the second condition, the variables used to match treatment and control subjects included 
gender, race/ethnicity, special education service receipt, LEP status, FARM status in grade 5 or 
grade 8 for the middle school and high school component, respectively; and state assessment 
scale score performance in reading and mathematics in grade 5 for the middle grades-only 
participants, grade 8 for the 2013 graduates, and grade 7 for the 2008 graduates (differences in 
choice of grade level for middle grades assessment data resulted from differences in both data 
availability and achieving a satisfactory data balance between treatment and control groups). 

We used the R package Matchlt (Ho, Imai, King and Stuart, 2007) to perform the matching 
algorithms, which estimates a logit regression of the probability, or “propensity” of receiving the 
treatment. This propensity can be expressed as: 

Logit (P) = Log [P / (1 - P)] 

This is regressed on relevant background characteristics, and specifically, the algorithm to 
identify “nearest neighbor” matches was estimated. A post-estimation propensity score, or 
probability of receiving the treatment for which each treatment case is assigned P( 1), is 
calculated for each case within the pool of potential control subjects, and based on an iterative 
estimate of the average propensity across potential control cases, suitable control subjects are 
identified and remaining subjects are discarded. In other words, the matching procedure 
determines the set of control students who, on average, are most similar to the treatment group. 

To ensure covariate balance, the standardized mean differences between treatment and the 
proposed control subjects for each matching variable were examined before further analyses 
were performed. The standardized mean difference is calculated by subtracting the control group 
mean from the treatment group mean and dividing the difference by the standard deviation for 
the control group. Standardized mean differences of less than .25 are considered appropriate for 
defining adequate covariate balance between groups (Rosenbaum and Rubin, 1983). 

For the current analyses, the number of control students was not always the same as the number 
of treatment students (i.e., if there were insufficient suitable control subjects for a one-to-one 
match for each treatment case), so weights generated in the matching procedure were applied to 
all further comparison analyses in these instances, where the weight for each treatment student is 
equal to 1, and the weight for any control student is equal to the inverse of the calculated 
probability score, balanced across the control cases so that the sum of the control weights 
amounts to the total of the number of treatment cases. This procedure allows for estimating the 
average effect of the treatment on the treated. 
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Appendix D 



Survey Constructs, Corresponding Questions, and Reliability Alphas 

Cronbach’s Alphas for: 

Construct 

Survey Questions High School 

Alumni 


“I can relate to the people around me in math and science 



Belongingness within the 
STEM field 

courses.” 

“I have a lot in common with the other students in my 

.90 

.82 

(l=Strongly Disagree; 
7=Strongly Agree) 

math/science courses.” 

“The other students in my math/science classes share my 
personal interests.” 




“A degree in STEM will allow me to obtain a job that I 
like.” 



Expectations for personal 
satisfaction with a career in 

“A degree in STEM will give me the kind of lifestyle I 
want.” 



STEM 

“I will feel ‘part of the group’ on my job if I enter a STEM 

.91 

.89 

(l=Strongly Disagree; 
7=Strongly Agree) 

field.” 

“A degree in STEM will allow me to get a job where 1 can 
use my talents and creativity.” 

“Doing well in math will increase my sense of self-worth.” 




“Someone like me can succeed in a math or science career.” 
“Doing well at math will enhance my career/job 



Expectations for success in 
STEM career 

opportunities.” 

“A degree in STEM will allow me to obtain a well-paying 

.82 

.76 

( l=Strongly Disagree; 
7=Strongly Agree) 

job.” 

“I will be treated fairly on the job. That is, I expect to be 
given the same opportunities for pay raises and promotions 
as my fellow workers if I enter a STEM career.” 




“I can complete the math requirements for most STEM 
majors.” 

“I can complete the science requirements for most STEM 



STEM self-efficacy 
(l=Strongly Disagree; 
7=Strongly Agree) 

majors.” 

“I can succeed in a STEM curriculum while not having to 
give up participation in my outside interests (e.g., 
extracurricular activities, family, or sports).” 

“I think I can succeed (earn an A or B) in science courses.” 
“I think I can succeed (earn an A or B) in math courses.” 

“I can complete a degree in STEM.” 

.87 

.89 

Self-Rating of 
Knowledge** 
(l=Major weakness; 

General Knowledge 

Knowledge of a particular field or discipline 
Understanding of national issues 

.73 

.90 

5=Major strength) 

Understanding of global issues 



Self-Rating of Creative 
Thinking Skills 
(l=Major weakness; 
5=Major strength) 

Critical thinking skills 
Problem-solving skills 
Ability to think “outside the box” 

Ability to think about a problem in a different way from 
most people 

.81 

.84 

Self-Rating of Soft Skills 
(l=Major weakness; 
5=Major strength) 

Ability to manage your time effectively 

Ability to get along with people of different races/cultures 

Ambition 

Leadership abilities 

.74 

.77 

**For the alumni, the Knowledge construct includes only ‘General knowledge’ and ‘Knowledge of a particular field 
or discipline’, while for high school students, the construct includes all four items. 
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